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Increasing student evaluation capacity through a collaborative 
community-based program evaluation teaching model 

Shauna K. Carlisle 1 & Jean M. Kruzich 2 

Abstract: The evaluation literature reflects a long-standing interest in ways to 
provide practical hands-on training experience in evaluation courses. 
Concomitantly, some funders have shown rising expectations for increased 
accountability on the part of Community-Based organizations (CBOs), even 
though agencies often lack the associated funding and expertise required. This 
paper describes a training evaluation model, developed and refined over the past 
ten years, that partners student needs for real-world evaluation experience with 
CBOs needs for technical expertise and assistance in conducting evaluation 
activities that support program improvement. The process of developing and 
implementing the model is presented, and content analysis of ten years of student 
feedback is summarized. 
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I. Introduction. 

In recent years the topic of nonprofit accountability has received increased interest and attention. 
In a statewide survey of North Carolina nonprofits, 61% of the 271 agencies surveyed indicated 
that evaluation activities had become more extensive, comprehensive, or rigorous in the past 
years (Murphy & Mitchell, 2007). At the same time research on nonprofits’ evaluation capacity 
suggests significant constraints to their ability to address these mandates (Carman & Fredericks 
2009). A study of evaluation practices in Canadian nonprofits found 37% identified a lack of 
internal capacity, such as staff and time, with 31% a indicating a lack of skills and knowledge in 
conducting evaluations as major challenges (Hall, Phillips, Meillat, & Pickering, 2003). With 
increased expectations on the part of some major funders, professional degree programs have 
seen an expansion in the market for skilled evaluators. Efforts to incorporate real-world 
experiences in program evaluation curriculum have been a focus across professional training 
programs including public health (Fierro & Christie, 2011; Davis, 2006) social work (Jacobson 
& Goheen, 2006; Anderson, 2003; Margolis, Stevens, Laraia, Ammerman, Harlan, Dodds, & 
Pollard, 2000; Lundahl, 2008) and education (Gredler & Johnson, 2001; Kelly & Kaczynski, 
2008). 

The most common methods used for providing practical hands-on training experiences 
include simulations, role-playing, course projects, and practicums (Trevisan, 2002; Trevisan, 
2004, Lee, LeBaron Wallace, & Alkin, 2007; Gredler & Johnson, 2001). However, few articles 
both describe a practical training approach and assess student responses to it. Further, no model 
was identified in the literature that had been implemented in both graduate and undergraduate 
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level courses. Therefore, this article has two goals. First it aims to describe a pedagogical 
model called Collaborative Community-based Program Evaluation (CCPE) that responds to the 
need for nonprofits to engage in evaluation activities while at the same time providing the real 
world experience and training students need. Second, this study also provides preliminary 
findings from an analysis of students’ responses to CCPE as a model for teaching graduate and 
undergraduate students’ program evaluation. 

CCPE is based on university level course instructors and students partnering with 
community-based agencies on evaluation projects that support students’ learning critical 
evaluation skills essential for meeting the demand for nonprofit evaluators. The goals are to 
create a mutually beneficial collaboration that increases both agency’s evaluation capacity and 
students’ individual evaluative capacity, and to increase student motivation, interest, and 
evaluation skills through working with a “real world” community partner. Content analysis of 
student responses to course evaluations from 12 graduate and 7 undergraduate courses was 
conducted to assess CCPE’s utility in preparing students for real world experiences in the field of 
evaluation and its effectiveness at increasing student interest in program evaluation research. 

II. Background and Significance. 

Hatry, Wholey, and Newcomer (2010) identify two primary reasons for evaluation activities: 1) 
to achieve greater accountability in the use of funds and 2) to improve the effectiveness of 
programs. While evaluation scholars and agency administrators view improving agency program 
effectiveness as the more important of the two (Cannan, 2010; Ebrahim, 2005; Wholey et al., 
2010) the literature suggests that most agencies have a primary focus on mandated external 
reporting to funders (Benjamin, 2012)—activities that often do not lead to program 
improvements. In a survey of 178 nonprofits, Cannan (2007) found that 8 of every 10 
community-based organizations reported reviewing programs and assessing whether they were 
meeting program goals and objectives (accountability), but far fewer organizations were engaged 
in evaluation and perfonnance measurement activities with 41% using a performance 
measurement system and 65% conducting a formal program evaluation of any of their programs. 
A second study reaffirmed this focus with executives identifying the most frequent use of 
evaluation being for accountability purposes, including reporting to Hinders (87%) and 
reporting to the board (84%) (Murphy & Mitchell, 2007). These discrepant priorities lead to 
nonprofit administrators preferring formative evaluations that help them know how to better 
operate their programs, while funders want summative evaluations that identify, “What did we 
cause?”, the answer to which does not improve agency services (Snibble, 2006). 

While nonprofits recognize the importance of measuring program outcomes, they face 
two significant hurdles. First, they lack the financial flexibility or support for evaluation 
activities. Many funders require program evaluations but rarely provide funds or grants to collect 
this information (Carman, 2007). Hall et al. (2003) found less than half of the 322 non-profit 
funders surveyed reported providing separate funds for evaluation or allocating project funds for 
evaluation purposes. A second hurdle is that agency executives and their staff may have limited 
or no formal training in evaluation and yet are expected to collect and report 
performance/evaluation data to stakeholders (Christensen & Ebrahim, 2006). The most frequent 
barriers to conducting evaluation activities reported in a sample of 271 agency executives were 
lack of funding (53%), insufficient staff (45%), and lack of affordable technical assistance (40%) 
(Murphy & Mitchell, 2007). 
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As a result, the agency’s need for technical assistance and program evaluation resources 
provides a real world opportunity for emerging practitioners to engage in CCPE. Through the 
course students develop technical and analytic evaluation competencies as well as interpersonal 
skills needed for conducting evaluation studies (communication, negotiation, conflict, 
collaboration, and cross-cultural sensitivity), topics seldom addressed in evaluation curriculums 
and even more rarely provided opportunities to apply. Further, this model provides entry-level 
evaluation training that includes the professional interpersonal skills needed to interact with 
clients and stakeholders, skills that have been identified as key competencies desired by 
employers but lacking in existing evaluation training (Dewey, Montrosse, Schroter, Sullins, & 
Mattox, 2008; Simons & Cleary, 2006; Taut & Alkin, 2003). The next section describes the 
CCPE approach, created and refined over the past ten years, followed by preliminary analysis of 
students’ and agencies’ responses to the collaboration experience. 

ITT. The Collaborative Community-Based Program Evaluation Model. 

A. Steps to Developing a Collaborative Community-Based Program Evaluation. 

Collaborations represent a mutually beneficial and well-defined relationship entered into by two 
or more organizations to achieve common goals. Essential characteristics of collaboration 
include: 1) joint action for mutual benefit; 2) interdependence and reciprocity; 3) mutual 
authority; and 4) shared responsibility, risks, resources, and rewards (Mattessich, Murray-Close, 
& Monsey, 2001). To maximize the likelihood of a successful collaboration the instructor needs 
to invest a significant amount of time prior to the course, developing a relationship with an 
agency partner identifying both parties’ respective interests, concerns and available resources, as 
well as negotiating roles and responsibilities with the goal of developing a set of group projects 
that are appropriate in size and scope for the time available. Based on past experiences teaching 
the course, the instructor developed a list of criteria that identifies the resources and 
commitments an agency needs to provide in order to provide a good fit with the goals of the class 
(See Appendix 1). 

While the criteria list is used to guide agency choice for both undergraduate and graduate 
courses, there are differences in emphases. At the graduate level, there is more of a priority to 
work with agencies that have existing data that can be analyzed in the multiple SPPS computer 
labs, since a primary goal is to develop the professional competencies of MSW students in the 
program administration and policy concentration. For the undergraduate class, the primary 
objective is to develop general research methods and data collections skills within the context of 
program evaluation that undergraduates apply to profit and nonprofit agencies. For both courses, 
the objective is to engage students in hands on “real world” experiences that simulate the 
experiences they will encounter within an occupational setting. Typically only one agency is a 
collaborator in both the undergraduate and the graduate class. The graduate class is capped at 25 
students, whereas the undergraduate class is typically capped between 32 and 45 students. On 
two occasions two agencies have been used for the undergraduate class, to ensure enough 
evaluation questions for students to address. 

A combination of methods are used to identify potential community agencies including 
recommendations from agency staff, board members, funding organizations, students with 
experience working or volunteering in the program, faculty colleagues, web searches, and when 
available, a university’s service learning program or community-based learning department can 
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be a valuable source. One or two quarters before the class the instructor contacts a potential 
agency to detennine if the agency has program evaluation needs they are unable to address with 
their current evaluation capacity and that might be appropriate for the given class. The instructor 
communicates to the agency the purpose of the course, the number and educational level of the 
students involved, course learning objectives, and provides a brief description of the CCPE 
process. The instructor emphasizes that a utilization-focused evaluation approach (Patton, 2008) 
is used whereby the instructor and students provide support and consultation in addressing the 
agency’s program evaluation need. Furthermore, the instructor explains that CCPE provides an 
opportunity for students to gain real world skills at the same time that the agency receives a 
quality program evaluation that will have utility for the agency. If, following this initial phone 
conversation, agency staff express interest in further exploring collaboration and the instructor 
believes there is a good likelihood that the agency would be a good match, an initial meeting is 
scheduled at the agency. Conducting the meeting at the agency allows the instructor to get a 
better sense of the agency size, programs, how its location may influence student access, check 
availability of meeting space for interviewing and an opportunity to be introduced to other 
agency staff. 

A set of questions, derived from the criteria for agency selection, are used to structure the 
initial meeting (See Table 1). These questions identify the instructor’s interests in the 
collaboration but also provide a vehicle for finding out what commitments and resources the 
agency is expecting from the class. The purpose of the first planning meeting is to identify the 
program evaluation interests, questions, and how the agency plans on using the program 
evaluation research. An important question the instructor asks the agency representative is “What 
final products are needed for this collaboration to be considered a success by the agency?” The 
agency’s ability to respond to the question helps guide the process. While program staff are often 
familiar with gathering information on program inputs and outputs and tracking data for billing 
purposes, they often have little experience with process evaluation and thinking about how 
research could be designed to improve the quality of their program. In some cases program staff 
may not have a college background or familiarity with research methods. This means the 
instructor needs to play an active role in helping shape a general staff request such as, “We want 
to find out if youth attending our arts and community service based program are more involved 
in their community following the program,” and tease out what specific types of data would be 
most useful, their rationale for wanting information, and helping identify the range of options for 
data collection and measurements choices. Taking along copies of student group reports from 
past classes is helpful in giving examples of different types of data collection methods and 
presentation of findings in addition to demonstrating the quality of student work. 

Table 1. Questions for Preliminary Community Agency Meeting. 

1. What questions are important to you as an agency? 

2. How many clients do you serve on an annual basis? 

3. What data is currently available for analysis? 

4. How will you use the findings? 

5. What methods for collecting data from clients would be pennissible? 

6. Will the director and program staff assist in the coordination of data collections 
between class and clients? 

7. Does your program have a logic model? If not would you be willing to work with 
the instructor to draft one? 
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8. Does this partnership have the approval of the agency leadership? 

9. Does the agency agree to pay incentives if needed to help recruit client 
participants? 

10. Will the director and contact person agree to attend the second and last class session? 

11. Is the agency an organization that serves marginalized communities and do they 
lack the capacity to evaluate the questions they have identified? 

Following the initial meeting the instructor creates a summary that is distributed to all 
meeting participants of research questions and possible projects the group discussed and any 
guidelines they agreed to on how they will work together. Participants are asked to review the 
summary and make any revisions so they can ensure the document represents shared 
commitments and understandings that will guide the partnership. Often times a second meeting is 
needed to further refine and specify the questions that will form the basis for the student projects. 
By the end of this iterative process, through meetings and email exchanges, the agency and 
instructor have clearly outlined in writing the specific projects that will be available for class 
members to choose from and the processes that will guide the work. The project list is distributed 
to students at the first class session. The CCPE group project is the major assignment and 
provides a focus for sequence of topics and activities for the ten-week quarter (see Appendix 2, 
which also includes recommended content for a semester length class). Early on in the course, 
the instructor shares with the students the process involved in engaging the agency and clarifying 
their information and program evaluation needs as well as the agreements that will guide the 
collaboration. 

For the graduate class these projects fall within 3 research methods categories: survey, 
focus groups and/or interviews, and secondary data analysis. For the undergraduate class, the 
agency and instructor identify the agencies key knowledge and program evaluation needs with 
the goal of implementing a data collection strategy. This can include program evaluation, model 
comparisons, outcome assessment, and descriptive and explanatory evaluation questions. At the 
undergraduate level at least 1 data collections strategy must be a survey in order to introduce 
students to survey research methods. Initial experiences teaching the undergraduate and graduate 
course made clear that having a logic model prior to the start of the course was of critical 
importance for any program we were evaluating. A logic model provides a visual illustration of 
the logical linkages among program resources, activities, outputs, clients served, and short, 
intermediate and longer term outcomes (McLaughlin & Jordan, 1999; W.K. Kellogg Foundation, 
2004; Knowlton & Phillips, 2009). If the partnering agency does not have a logic model for their 
program to be evaluated, the instructor for the undergraduate or graduate course co-creates one 
with program staff. Having a program logic model serves a number of purposes. First, a 
program logic model helps ensure there is a shared understanding on the part of the program staff 
and class members on how the program operates and this is one way to help guarantee that 
course deliverables are helpful to agencies. Second, because a logic model visually depicts key 
aspects of the program it is a strong tool for communicating with diverse stakeholders- those 
who have varying world views and different levels of experience with program development and 
evaluation; an important consideration when working with small, community based 
organizations. In the undergraduate course, where there are often four group projects with eight 
program evaluation questions pursued by the class, a logic model helps organize possible 
research questions identified by students and is useful in focusing the research questions and 
projects that students generate. 
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Prior to beginning the class projects, the instructors follow the same human subjects 
procedures used in research projects to attain IRB approval. The section of the Federal Register 
that pertains to the Department of Health and Social Services Protection of Human Subjects 
defines research as, “a systematic investigation, including research development, testing and 
evaluation, designed to develop or contribute to generalizable knowledge” (Human Subjects 
Protection, 2009). Because the program evaluation activities we engage in with a community 
partner are for the agency’s internal purposes, are not designed to add to generalizable 
knowledge, and will not be published, they are not considered research under Part 46.101 of this 
regulation (Human Subjects Protection, part 46.101, 2009). In spite of the fact that our evaluation 
activities are exempt and so do not require IRB Board Review, course instructors follow the 
protocols that govern research that is reviewed by IRBs including participant consent forms and 
protection of confidentiality. 

B. First Day of Class. 

Once projects have been identified, the instructor in both the undergraduate and graduate course 
meets with students on the first day of class and shares with them: information on the program 
the students will be evaluating, 1) background on the agency, 2) summary statistics from the 
agency, 3) population served by the agency, 4) programs offered by the agency, and 5) contact 
information for the agency. Often this infonnation is shared in a packet format, and students are 
given a short period of class time to review the material and course syllabus. The instructor 
describes the activities carried out the prior quarter to begin building the agency/class 
collaboration, how the instructor identified the agency partner, steps involved in developing a 
relationship with the program director and processes that culminated in the list of group project 
topics distributed during the first class session. Students also sign a confidentiality agreement 
that prohibits sharing any client or staff information gathered during the evaluation without 
authorization. The agreement states that a violation of confidentiality will result in a failing grade 
for the course. 

While there are many similarities in the implementation of the CPPE model across 
educational levels, there are some notable differences. Students enter the graduate level course 
having completed a statistics and introductory research methods course. Following a brief 
overview of the history, services provided and populations served by the agency program 
partner, each of the possible group projects are discussed in detail. Once students have a clear 
understanding of the focus of each project they self-select themselves into one of four or five 
groups based on their interest in the topic and/or the data collection method. Project options 
always include a range of research methods so that students can pursue their interest in 
quantitative or qualitative methods. After class members have chosen from among the project 
topics, they meet in their project groups to share contact infonnation, available times for meeting 
outside of class, formalize ground rules for how the work group will operate, and identify a 
group member who agrees to act as the group’s liaison with the project staff identified as the 
agency contact for that group roles. They also develop a list of questions to ask agency staff who 
will attend the first half of the upcoming second class session. 

The undergraduate research class is an introduction to research therefore a detailed 
outline of the project is provided by the instructor that includes two very broadly written research 
questions per project. Students must decide which specific program evaluation project and 
question most aligns with their interests. Most students identify a project and question relatively 
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easily. During the second half of the class (after students have identified their project and 
question), they meet in their groups to develop a group contract and resume. The group contract 
and resume is designed to insure that all members understand their roles and responsibilities and 
to identify the unique skills each member brings to the group. Students are asked to respond to 
four questions as a group: 1) What are the individual strengths or skills in our group? 2) What 
type of communication methods will we use? 3) What are the group expectations around 
participating in group meetings, timelines, and deadlines? 4) What role will each member fulfill 
(facilitator, project manager, editor, etc.). This infonnation is then written in the form of a 
contract, signed by each member of the group, forwarded to the instructor and posted on the 
group Blackboard discussion board. This is especially important given the enonnity of the 
project for undergraduate students and the need to clearly outline group participation 
expectations. Students are then asked to develop interview questions for a session with the 
agency partner. The instructor delivers an online group assessment four times throughout the 
quarter that allows students to provide feedback on their group member’s participation. The 
feedback on this survey is used by the instructor to insure that each member is participating fully 
and to guard against social loafing. 

C. Agency Interview and Class Sessions. 

An introduction to the agency partner occurs during the second-class session in both 
undergraduate and graduate classes. Agency program staff attend the class to 1) meet the 
students with whom they will be interacting, 2) describe in person their agency, services, 
constituents, etc. 3) describe how they intend to use the evaluation report findings and 4) 
describe why student’s projects are important to the functioning of the agency. Students then 
have an opportunity to pose questions to program staff that were not addressed in the agency 
information packet provided during the first class session. This class session is critical for 
engaging students in the process. By the end of the session students identify feeling a stronger 
sense of responsibility and commitment to the final products. 

For both undergraduate and graduate courses, class time includes a combination of 
lectures, group discussion, computer labs, and project group work. Working in small groups, all 
class members conduct evaluation activities with the same agency program. A utilization- 
focused evaluation perspective with its emphasis on addressing specific, practical needs of the 
users of the evaluation, and joint control over the research and ongoing participant involvement 
undergirds both courses (King, Cousins, & Whitmore, 2007; Patton, 2008). For the graduate 
course, project groups may focus on specific aspects of the evaluation process, such as 
developing and piloting a measure or analyzing existing pre/post data. The focus is on 
development of specific knowledge and skills regarding important evaluation relevant topics 
including conducting a content analysis, completing an evaluation program audit of evaluation 
practices at their practicum agency, statistical analysis of quantitative data and responding to 
challenging ethical scenarios 

In contrast, the undergraduate course has all students engage in all stages of the program 
evaluation process, including identifying relevant theories, conceptualization and 
operationalization of variables, development of research questions, survey design, data 
collections, data cleaning, storage, and analysis, and dissemination. Instruction for 
undergraduates is based on a progressive model where students first receive a lecture on the 
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concept and then engage in a group activity to reinforce the concept. This process is repeated 
until all the research steps are covered and a final report is provided to the agency. 

D. Agency Presentations. 

In both undergraduate and graduate courses students completed a final written report, and to 
develop a 15- to 20-minute formal PowerPoint presentation of their major findings and 
recommendations that is to be presented on the last day of class. Members of multiple agency 
stakeholders groups often attend including agency executives, board members, program staff, 
volunteers, and program participants. Stakeholders have an opportunity to ask questions of the 
students and make comments on the findings. The group Power Point presentation includes 
identifying the project group’s evaluation questions, variables of interest, data analysis approach, 
significant findings, limitations of the analysis, and suggestions for utilizing the project findings. 
Students receive verbal feedback from the agency and verbal and written feedback from the 
instructor on the presentation and written report. Each student group provides the agency a copy 
of the PowerPoint, a written evaluation report, covering all aspects of the evaluation process as 
well as an executive summary and data analysis details so that agencies can replicate student 
research findings. To date undergraduate and graduate classes have produced over 25 projects 
through partnerships with 12 community agencies. These projects include instrument 
development, needs assessments, assessing service delivery, comparing service delivery models, 
customer satisfaction surveys, and outcome evaluations. Table 2 includes a list of typical 
projects. 

IV. Evaluating the Success of Collaborative Community-Based Program Evaluation. 

A. Methods. 

A university wide Instructional Assessment Form that includes open-ended questions was 
distributed to students on the final class day. To identify what aspects of the course were most 
important to students, responses to the question “What aspects of this class contributed the most 
to your learning?” were transcribed and content analyzed from 12 graduate level and 7 
undergraduate level courses held between 2000 and 2011 (533 students). To assess inter-coder 
reliability, the degree to which coding of text by two coders were similar, the co-authors 
independently examined 15% of all responses, drafted a set of emergent categories, then met to 
compare proposed codes and agree on an initial master list of codes. After the initial draft 
codebook was developed, an iterative process of coding, reliability assessment, codebook 
modification, and recoding was used. After three coding rounds, inter-coder reliability was 
conducted to validate the coding scheme. Using SPSS, Cohen’s kappa (Cohen, 1960) was 
calculated resulting in a kappa of.85, an excellent level of inter-coder reliability (Lombard, 
Snyder-Duch, & Bracken, 2002). Each author then coded half of all student responses. 

V. Results. 

Table 3 summarizes the results of a content analysis of 553 student responses to the question 
“What aspects of this class contributed the most to your learning?” The unit of analysis is a text 
segment; therefore, the sample size (n) represents the number of text segments coded for each 
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row category by the total graduate sample and undergraduate sample. Of the four overarching 
categories, professional application represented 234 of the 553 (42.3%) of the coded segments. 
This is a good indicator of the importance of the CCPE model. 

Table 2. Illustrative Evaluation Projects. 

• Assessment of an educational model designed for students with family members 
receiving treatment for life threatening illnesses. 

• Assessment of the utility of an early childhood education brochure. 

• Evaluate the impact of agency programming on youth's feelings of empowerment and 
identity, use of time, values and risk behaviors. 

• Six month follow up telephone interview with summer camp attendees. 

• Analysis of pre and posttest results of camp program serving marginalized youth. 

• Pilot evaluation of mentors' experience participating in a positive youth development 
program. 

• Development and pilot testing of caregiver satisfaction measure for a community-based 
youth counseling program. 

• Assessment of prescription medication and medical care use by homeless youth. 

A. Professional Application 

Real world hands-on experience was a category of professional application mentioned frequently 
by graduate (46%) and undergraduate (38%) students: 

“I loved the applied (experiential) learning. Plus it makes me feel more connected to the 
[JW when I can conduct meaningful research. We’re a research institution are we not?” 
(Undergraduate) 

“The actual applied hands-on learning was amazing, it facilitated the most challenging, 
but educational experience I've had by far”. (Graduate) 

“Learning how ’real’ processes play out in the ‘real’ world”. (Graduate) 

While working as part of a research team was an important factor for some students at 
both educational levels, undergraduates mentioned it more than twice as often (43%) as did 
graduate students (17%). Responses were coded as “Part of a research team” when a student’s 
response focused on the process or interactions within their team. While the most frequent 
comment was simply, “working in” or “group work,” one student contrasted the experience with 
her other courses, “Group work. It makes the class interesting and on your toes! It is a unique 
class, much different than other classes this class made me feel independent by working on group 
project. “Two undergraduate students provided a bit more detail that hints at the challenging 
nature of the project: “Group experience was good. We could help each other muddle through,” 
and “Group work was most helpful because I had a social support network.” While 
undergraduates stressed the support provided, a graduate student’s comments, “Group members’ 
strengths and ability to teach and share what they know,” suggests peer learning may play a 
larger role at the graduate level where students are more likely to bring additional professional 
practice experience. 
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Table 3. Percentage and Frequency of Text Segments from Student Responses to the 
Question, “What Contributed Most to Your Learning?” _ 



Total 

Sample 

Total 

Sample 

Graduate 

Graduate 

Undergrad 

Undergrad 

Coding Categories 

n 

% 

n 

% 

n 

% 

Professional Application 

234 


128 


106 


Real World/Hands-on 

99 

43 

59 

46 

40 

38 

Work with Agency Partner 

32 

14 

23 

18 

9 

8 

Increased Confidence 

15 

6 

12 

9 

3 

3 

Increased Interpersonal Skill 

1 

0 

1 

1 

0 

0 

Highly Motivating 

11 

5 

4 

3 

7 

7 

Transformational Learning 

6 

2 

5 

4 

1 

1 

Part of Research Team 

68 

29 

22 

17 

46 

43 

Increased Presentation Skills 

2 

1 

2 

2 

0 

0 

Teaching Strategies 

113 


73 


40 


Course Structure 

35 

31 

16 

22 

19 

48 

Group Presentation 

14 

12 

12 

16 

2 

5 

Lecture 

19 

17 

5 

7 

14 

35 

Class Discussion 

17 

15 

13 

18 

4 

10 

Guest Presenter 

7 

6 

7 

10 

0 

0 

Course Materials 

21 

19 

20 

27 

1 

3 

Class Assignments 

109 


65 


44 


Data Analysis (Qualitative & 

41 

38 

29 

45 

12 

28 

Quantitative) 







Group Project 

68 

62 

36 

55 

32 

72 

Instructor Characteristics 

97 


79 


18 


PE Expertise-Experience 

9 

9 

8 

9 

1 

5 

Knowledge 

6 

6 

5 

5 

1 

5 

Responsiveness 

33 

34 

30 

33 

3 

14 

Accessibility 

21 

23 

18 

20 

3 

14 

Instructor Attitude 

15 

13 

10 

11 

5 

24 

Teaching Skill 

13 

12 

8 

9 

5 

24 


B. Teaching Strategies. 

Of the four categories, teaching strategies showed the most variation between graduate and 
undergraduates students’ perspectives of what contributed most to their learning. A course 
structure that closely integrated lecture and application for each step in the evaluation process 
was more often important to undergraduates (48%) as compared to 22% of graduate students: 

“I love the setup of the class, instead of lecturing the whole quarter we actually 
got to conduct our own study”. (Undergraduate) 

“Going step-by step with support. Able to ask questions (Undergraduate) 

“The instructor did a fabulous job breaking points down and applying it to our 
project”. (Graduate) 

Lectures, which generally play a larger role in introductory undergraduate 
classes were also key to undergraduate students learning (35%) while much less 
so for graduate students (7%). What was most important to graduate students in 
tenns of delivery methods were course materials including the program 
evaluation text, handouts, and required supplementary articles on different 
evaluations studies (27% compared to 3% for undergraduate). 
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C. Class Assignments. 

While important to both groups, undergraduates (72%) more often noted the importance of 
carrying out of the program evaluation as compared to graduate students (55%). It may be that 
the CPPE model supported developing higher level cognitive skills than is usually found in 
undergraduate courses (Anderson & Krathwohl, 2001), a possibility suggested by a student’s 
remark, “The group project forced us to apply what we learned and not just regurgitate info 
during a test.” Graduate students (45%) were more likely than undergraduates (28%) to remark 
on the value of the data analysis assignments. 

To ensure all graduate students had experience with different types of analysis, a 
qualitative coding exercise, in addition to computer lab exercises that required using SPSS with 
an existing sample of the agency’s data were assigned early in the quarter to give students 
practice in analyzing data prior to gathering their project data. Illustrative student comments in 
response to the question of what aspects contributed most to your learning included, “The hands- 
on practice of SPSS” and “Being in the computer lab and using SPSS where we were able to get 
one-to-one help” as well as “coding qualitative responses”. 

D. Instructor Characteristics. 

With only 10 weeks to complete their evaluation projects students felt under considerable 
pressure for fast turnaround and often requested consultation with the instructor or feedback 
from an agency liaison on a weekly basis. For graduate students, the instructor characteristics 
mentioned most frequently as contributing to their learning were responsiveness to student needs 
(33%), with comments such as, “following up with students to ensure understanding” and 
instructor accessibility (23%) including meeting outside of class. A student commented, 
“Instructor spent many hours with us outside of class helping us to figure things out;” and 
another wrote, “Knowing that the professor was there to answer questions.” Undergraduate 
students most often remarked on the instructors’ attitudes (24%) in terms of her “enthusiasm” or 
“energy” and teaching skills (28%) particularly her excellence in communicating and attention to 
different ways of learning. 

D. Agency Responses. 

Systematic data on agency partners’ experience with the class has not been collected. However, 
some comments, which are uniformly positive, from agency staff, executive and board members 
have been saved by the instructors over the last 10 years. Agencies frequently request to continue 
working with the class during the upcoming academic year and increasingly the instructor is 
working with the same agency for two consecutive classes, offered once per year to further 
strengthen their evaluation capacity. Comments from agencies illustrate that the work students 
produce is not only beneficial for program development but for accountability as well. An 
agency commenting on the benefit of the shared work noted. “The results of your work will have 
great impact! They will be reported on our website and to community leaders and partners. Your 
recommendations will guide our decisions on scaling and enhancing the project next year.” 
Another comment highlighted the mutual value of the collaboration: “Thank you for partnering 
with (the Agency) on our (the Project) work. I think both sides really gained from the experience. 
I have a nifty report with very good data that I can use to target our efforts.” 
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A Clinical Director of a multi-service center remarked at the conclusion of the course that 
the benefits of the collaboration extended beyond the evaluation report: “I’ve come to appreciate 
how this program speaks to something far greater than just the individual tasks undertaken by the 
students. It’s an alliance between a community’s service providers and the educational system 
that prepares them. Connecting all this focused academic energy with real — and somewhat 
overwhelming — issues within the community offers multiple benefits. The students get a 
chance to see some things that are likely not covered in their textbooks, and we get access to an 
incredible amount of intellectual power, energy, and research tools that we could never afford on 
our own.” In addition to the instrumental use conveyed in the prior statement, involvement in an 
evaluation may lead to process use, changes in the thoughts and behaviors of individuals, which 
then impacts organizational change (Preskill, Zuckerman, & Matthews, 2003). An agency 
partner, commenting on the partnership captures both: “It has left us with practical resources that 
are already positively impacting the families we serve. From opening up a dialogue with families 
through the survey to finding ways to get better information from our data, we’re right at the 
genesis of something that feels more structurally sound than what we had before.” 

After a decade of offering the course, former students, who are now program directors 
and board members of nonprofit agencies frequently ask if their agency can become a site for 
partnering with the class. Fonner students recognize the significant contribution CCPE classes 
can make in preparing students for program evaluation and for helping their agencies achieve 
their program development and accountability needs. 

VI. Conclusion. 

The purpose of this article was twofold; to describe the CCPE pedagogical model in sufficient 
detail that instructors interested in conducting a collaborative community based program 
evaluation course could confidently implement the model and second, to examine data gathered 
over the past decade from student perception of CCPE’s impact on their learning. We conducted 
a content analysis of course evaluations from twelve graduate level and seven undergraduate 
level courses, held between 2000 and 2011, focusing on the question “What aspects of this class 
contributed most to your learning?” Results of this analysis indicate that both undergraduate and 
graduate students overwhelmingly identified that the aspect of CCPE that contributed most to 
their learning was the real world experiences, followed by working on a research team. These 
findings show that students believed this pedagogical model was useful in preparing them as 
evaluators. Further, the findings reveal that working with a community partner was both 
engaging and fueled their motivation for completing the class work. 

Because we saw the enormous amount of work students put into their evaluation projects, 
far surpassing the time commitment and energy seen in other courses and considering that 
students are usually unhappy with the standard lecture based research classes, we expected more 
comments referring to the course as highly motivating. Typical responses included a graduate 
student who remarked, “I love that we were working with real data for a real agency. This made 
my learning and work seem way more relevant and really motivated me to do my best,” and 
another mentioned, “I was more invested knowing that we were positively contributing to an 
organization.” As we reviewed the individual comments we realized that the nature of the 
question, “What contributed to your learning?” pointed students toward mentioning different 
dimensions of the class rather than student attitudes. That is, an individual’s motivation may 
have been articulated in their desire to act on behalf of the agency and therefore coded as the 
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working with an agency partner. Despite the limited textual support for motivation, CCPE, as a 
pedagogical model, contributed to students viewing themselves as being well prepared by real 
world experiences. 

We would be remiss if we did not note the increased demands on the instructor 
implementing this approach to teaching collaborative community based program evaluation. 
Adopting the CCPE model significantly increases the demands on the instructor when compared 
with more traditional courses that do not involve ongoing community agency collaborations. For 
example, providing students with real world evaluation experiences requires preliminary work 
identifying and working with a community partner, meeting with students regularly, juggling the 
teaching of theory with practice, reviewing many drafts of a final product and ensuring the 
program evaluation needs of the agency are addressed require time above and beyond that of a 
traditional course. Additionally, faculty using this model must anticipate dedicated supervision 
and planning time over and above what is required for other courses. This is critical when 
multiple course projects must be completed within an academic term. One approach to help 
minimize some of the additional time demands on the instructor is to work with the same 
program for two years so that the upfront time on engagement, relationship building with staff, 
and becoming familiar with the program’s data collection systems are activities that do not need 
to be repeated during the second year of working together. Agencies usually request to work with 
the instructor for another year since their evaluation efforts can benefit from additional support 
and it provides an opportunity for additional kinds of analyses such as completing pre and post 
measures the second year using a measure developed during the first year of the collaboration. 
Additionally, given the emphasis on university/community partnerships the first author has 
received support for a TA to assist in the data collections and data analysis process in the 
classroom. Universities may find providing funds for a teaching assistant a good investment to 
provide support for community based learning. 

This study is not without its limitations. Given the preliminary nature of these findings, 
gathering additional quantitative infonnation on students’ perception of the utility of this 
approach as well as their feedback on how to strengthen the course would be helpful. Secondly, a 
comparison of the outcomes of this model to the outcomes of a standard research course would 
provide a more comprehensive understanding of the impact of this approach on student learning. 
Thirdly, asking our community partners to provide structured feedback on their experience of the 
collaboration and ways they intend to use the results would help with further developing the 
curriculum. Lastly, following up with community partners a year after the end of the class to see 
if they did use the results as they had intended and if there were other ways in which our 
collaboration had influenced the program’s attention, focus and involvement in evaluation 
activities will add to our understanding of factors that influence the use of evaluation results, and 
development of program’s evaluation capacity. 
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Appendices 

Appendix 1. List of Criteria Used for Selection of Agency Partners. 

1. Agency serves a minimum of 30-40 clients annually so there are sufficient 
numbers to calculate statistical significance. 

2. Preferable if agency has an existing data set that has not been analyzed that can be 
used in class computer labs so that students receive experience in quantitative 
analysis using SPSS with the programs’ data. 

3. Agency is interested in evaluation questions that include qualitative and quantitative 
data collections strategies or a combination of both. Multiple data collections methods 
are used so that students can gain experience with different approaches. 

4. Program has a logic model or program leadership is willing to develop a draft logic 
model prior to the quarter when the course is taught. 

5. Agency executive is interested and supportive of evaluation partnership and informs 
board of the collaboration. 

6. Agency staff is willing to serve as liaisons to student project groups and respond by 
email, phone, or in-person meetings with student groups during the quarter. 

7. Program director and staff commit to attend the second-class session, to share 
information about the program and answer students' questions and attend the last class 
session to hear student's PowerPoint presentations of their group project findings. 

8. Program staff agrees to help coordinate and set up schedule for data collection 
requiring interviews, focus groups, etc. with clients or staff. 

9. Agency agrees to pay for an incentive (providing food at focus group, fast food gift card; 
childcare, transportation), if that is needed to recruit client participants. 

10. Agency is a community based grassroots organization that lacks capacity to address 
evaluation questions they have identified. 
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Appendix 2. Course Activities and Topics for Ten Week 3 Undergraduate and Graduate 
Courses 4 . 



Undergraduate 

Graduate 

Week 1 

Overview of program/List of projects 

Sign confidentiality form. 

Students choose group project. 

Meet in groups to develop group resume & contract. 

Develop questions for program staff. 

Begin research on topic. 

Overview of program/List of projects 

Sign confidentiality form. 

Students choose group project. 

Meet in groups. 

Develop questions for program staff. 

Week 2 

Staff attend class, answer student questions. 

Lecture on levels of measurement. 

Begin literature review. 

Lecture on survey development. 

Groups begin identifying research questions and hypotheses. 

Staff attend class, answer student questions. 

Lecture on survey development. 

Week 3 

Develop survey questions. 

Finish draft 1 of literature, submit for feedback. 

Develop qualitative questions. 

Agency recruits research participants. 

Draft survey questions. 

Lecture on research design and sampling. 

Ethical and cultural concerns in PE. 

Discuss completed "Practicum Evaluation Audit" assignment. 
Agency recruits research participants. 

Week 4 

Send draft of survey questions to program staff for vetting. 
Forward draft research design and sampling method. 

Prepare for midterm exam. 

Send draft of survey/focus group questions to program staff for 
vetting. 

Qualitative Methods & Content Analysis. 

Computer session 1 & lab assignment. 

Week 5 

Based on agency and instructor feedback survey and qualitative 
questions are revised. Develop 

consent form. 

Forward survey to agency for dissemination. 

Based on agency and instructor feedback survey and qualitative 
questions are revised. 

Develop consent form. 

Midterm exam 

Week 6 

Lecture on data analysis. Lecture on data analysis. 

Lecture on SPSS. Computer session 2 & lab assignment. 

Begin data analysis. Implement focus groups/individual interviews or disseminate 

Submit draft research methods section to instructor for feedback, self-administered surveys. 

Lecture on evaluation designs. 

Computer session 2 & lab assignment including reliability 
analysis. 

Week 7 

Data analysis & interpretation. 

Develop tables and graphs. 

Develop results section. 

Data analysis & interpretation. 

Input data, transcribe interviews. 

Week 8 

Continue data analysis and results. 

Develop conclusion and suggestions. 

Submit part 3 of report for instructor feedback. 

Develop tables and graphs. 

Draft report and PowerPoint (PPT). 

Week 9 

Begin PowerPoint presentation development. 

Instructor feedback used to revise PPT presentation. 

Use instructor feedback to revise part 3. 

Submit full report for instructor feedback. 

Dress rehearsal of PPT presentation. 

Instructor & class feedback used to revise PPT presentation. 

Week 10 

PPT presentation to agency stakeholders. 

Final presentation feedback from instructor. 

Final group project reports given to staff. 

PPT presentation to agency stakeholders. 

Final presentation feedback from instructor and students. 

Final group reports given to program staff during week 
following last class. 
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Offering the course in a 15 or 16 week semester would allow students to more effectively integrate and consolidate their learning 
than is possible in a 10 week quarter. When offered on a semester basis, implementing the CCPE model would benefit from 
additional attention in the following areas: 1) alternative ways to approach literature reviews to strengthen different aspects of 
group projects, 2) review and critique of potential measures for groups projects, 3) practice and receive feedback on reporting 
written findings and interpretations, 4) discuss initial findings with agency program director prior to class presentation, 5) 
examine alternative methods to use graphs and figures 

to report results and 6) use class time for group project teams to share progress with other teams and provide consultation and 
peer review to each other throughout the course. 

4 Items in italics are topics or activities that vary between courses. 
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